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Abstract 

This  poster  describes  the  design  and  construction  of  a  world-class 
tiled-display  cluster  in  our  laboratory  space.  It  features  a  245.76 
megapixel  display  surface  (60  WQXGA  monitors)  with  a  270°  field 
of  view.  The  display  is  powered  by  a  18-node  rack-mount  machine 
cluster  located  in  an  adjacent  room.  We  will  summarize  the  sys¬ 
tem  design  and  provide  advice  for  anyone  interested  in  building  a 
similar  cluster. 

Index  Terms:  1.3.2  [COMPUTER  GRAPHICS]:  Graphics 

Systems — Distributed/network  graphics; 

1  Motivation  and  Design 

Regardless  of  the  resources  are  available  to  a  single  workstation, 
cluster  computing  remains  a  simple  means  to  theoretically  mul¬ 
tiply  computational  throughput.  In  the  same  way,  individual  dis¬ 
plays  or  projectors  can  be  combined  to  expand  the  visual  real  estate 
of  system.  A  tiled-display  cluster  is  a  combination  of  networked 
machines  and  multiple  monitors.  Tiled-display  clusters  allow  re¬ 
searchers  to  employ  these  resources  on  a  single  large-scale  visual¬ 
ization  or  series  of  visualizations.  Applications  include  immersive 
virtual  environments,  presentation  demonstrations,  group  collabo¬ 
rative  views,  and  command-center  views. 

We  endeavored  to  build  the  most  advanced  tiled-display  cluster 
possible  within  a  limited  laboratory  space.  We  examined  existing 
systems  at  other  institutions  and  this  informed  our  design  [5]  [3]. 
One  motivating  goal  was  to  replace  a  older  CAVE  (Cave  Automatic 
Virtual  Environment)  system  in  a  smaller  space  using  consumer 
hardware.  However,  we  were  also  moving  away  from  research  into 
virtual  environments  and  focusing  heavily  on  large-scale  VisAna- 
lystics.  Instead  of  one  large  view,  we  may  want  to  have  a  number 
of  separate  views  of  different,  equally  important  data  sets. 

For  these  reasons,  decided  on  high-resolution  desktop  monitors 
over  larger  format,  thin  bezel  displays  so  that  we  could  get  rea¬ 
sonably  close  to  the  displays  and  still  see  new  detail.  We  chose 
30”  WQXGA  desktop  monitors  as  the  tiles  for  our  display  wall 
(WQXGA  resolution  is  2560x1600). 

Figure  1  shows  a  visual  summary  of  our  design  process.  We 
choose  the  monitors  first,  developed  a  3D  sketch  of  how  the  mon¬ 
itors  would  fill  the  space,  commissioned  a  designer  to  design  a 
stand-alone  support  structure  for  the  monitor  wall,  and  placed  a  ma¬ 
chine  cluster  opposite  the  monitor  space  to  run  the  wall. 

The  end  result  was  an  18  node  cluster,  with  15  nodes  displaying 
to  a  60-monitor,  tiled-display  matrix  which  spans  three  walls  in  a 
U-shape.  The  remaining  nodes  are  used  for  auxiliary  computation 
and  coordination. 

The  wall  configuration  uses  a  free-standing  monitor-support 
structure  and  completely  fills  the  available  space.  The  cluster  it¬ 
self  is  stored  in  two  42U  server  racks  in  a  separate  room.  Bundles 
of  fifteen-meter  DVI  cables  run  through  the  ceiling  from  the  racks 
to  the  displays  in  cable  trays.  Each  rack-mount  workstation  houses 
two  graphics  cards  (Nvidia  Quadro  FX  5800),  and  each  graphics 


card  in  a  display  node  renders  two  tiles  in  the  wall,  for  a  total  of 
four  displays  per  node. 

We  collected  quotes  for  components  to  be  part  of  the  system  in 
March  2009,  but  it  took  two  years  before  the  cluster  would  be  com¬ 
pletely  operational.  Unanticipated  complexities  delayed  the  con¬ 
struction  and  testing  of  the  two  server-racks  of  workstation  nodes. 
We  also  needed  to  have  additional  power  and  cooling  capacity  de¬ 
signed  and  installed  into  the  space  for  the  wall  and  the  cluster. 


Figure  1 :  Left  to  right,  top  to  bottom:  A  3D  sketch  of  the  wall  lay¬ 
out,  the  support  structure  design,  a  photo  of  the  wall  display  Google 
Earth,  and  a  floorplan  of  the  laboratory  space  occupied  by  the  moni¬ 
tor  wall  and  the  cluster  that  runs  it. 


2  Application 

Since  installation  we  have  constructed  several  applications  to 
demonstrate  the  potential  of  the  system.  Figure  2  shows  a  portion  of 
the  wall  displaying  one  of  our  demo  applications.  Here  we  have  a 
spherical  panorama  of  the  night  sky  created  by  Axel  Mellinger1  [4], 
The  original  image  is  36000  pixels  by  18000  pixels.  It  is  broken 
into  many  smaller  images  and  displayed  as  a  pyramid  of  tiles  when 
displayed  in  the  web  application.  In  our  demonstration  of  the  wall, 
the  image  is  loaded  in  to  every  GPU  (Using  65%  of  the  available 
4GB  s)  and  displayed  as  one  large  view  using  the  Equalizer  frame¬ 
work  [1],  The  image  is  mapped  to  the  inside  of  a  sphere,  and  the 
views  from  each  tile  are  ordinated  to  look  in  the  outward  into  that 
sphere.  We  also  place  labeled  cursors  around  objects  of  interest, 
just  ask  Axel  Mellinger  has  done  in  his  web  application.  The  end 
result  is  a  highly-interactive  immersive  view  of  the  night  sky.  A 
user  can  use  a  mobile  device  to  communicate  with  the  head  node  to 
rotate  the  view. 

3  Lessons  Learned 

Over  the  course  of  the  design,  purchase,  construction,  and  instal¬ 
lation  of  this  resource,  we  learned  several  lessons  which  we  will 
describe  in  detail. 


‘e-mail:  mark.livingston@nrl.navy.mil  3.1  Design  with  Maintenance  in  Mind 

^e-mail:  jonathan.decker@nrl.navy.mil  We  constructed  a  tiled-display  matrix  which  makes  the  most  use  of 

the  available  space.  However,  this  has  made  maintenance  incon- 

1  Axel  Mellinger’s  Milky  Way  Panorama  can  be  seen  on  the  web  at: 
http://galaxy.phy.cmich.edu/~axel/mwpan2/kipano/ 


Figure  2:  A  photo  of  the  wall  display  a  large-scale  spherical 
panorama  of  the  sky  with  marked  objects 


venient.  We  currently  need  to  disassemble  large  parts  of  the  dis¬ 
play  support  structure  to  reach  cable  ports  for  troubleshooting.  The 
U-shape  of  the  display  gave  us  one  unexpected  benefit  for  mainte¬ 
nance,  in  that  the  curve  provides  a  small  crawl  space  in  the  “cor¬ 
ners”  of  the  display  that  would  otherwise  only  be  accessible  by  dis¬ 
mantling  the  wall  from  the  nearer  end  of  the  U-shape. 

3.2  Budget  for  Redundancy 

We  only  need  fifteen  nodes  to  render  the  display  tiles,  but  we  pur¬ 
chased  eighteen  nodes.  This  has  allowed  us  to  use  a  separate  ma¬ 
chine  as  a  master  node,  and  switch  components  to  troubleshooting 
problems.  Additionally,  we  recommend  purchasing  additional  dis¬ 
play  and  Ethernet  cables  in  case  of  failure. 

3.3  Avoid  Separating  Displays  and  Machines 

We  opted  to  rack-mount  our  workstations  early  in  the  design.  While 
this  provides  us  with  easy  access  to  the  nodes,  it  required  running 
long  display  cables  through  the  ceiling.  We  suggest  avoiding  the 
installation  and  maintenance  overhead  of  rack-mount  cluster  nodes, 
and  instead  placing  them  near  the  displays  with  easy  access  to  the 
back,  which  was  already  recommended  above.  Long  runs  of  display 
cables  are  expensive  and  difficult  to  replace  in  case  of  point  failure. 
The  best  approach  would  be  to  place  stand-alone  nodes  near  the 
displays  themselves,  and  to  leave  space  to  get  around  the  display 
wall.  Also  consider  that  newer  workstation  graphics  cards  have 
alternatives  to  bulky  DVI-D  cables,  such  as  DisplayPort  and  HDMI 
(see:  Nvidia  Quadro  6000).  We  recommend  these  formats  because 
they  are  much  easier  to  plug-and-play. 

3.4  Leverage  Open  Source  Frameworks 

We  have  explored  several  venues  of  multi-head  application  devel¬ 
opment,  including  a  from-scratch  approach.  We  are  currently  uti¬ 
lizing  Equalizer  [1],  a  GLUT-like  C++  framework  for  parallel  ren¬ 
dering  with  OpenGL.  We  started  out  using  Rocks  Cluster  5.4  [2], 
Rocks  installs  nodes  over  PXE  and  walks  you  though  tile-display 
configuration  with  its  ”Vis  Roll”  add-on.  However,  it  forces  you 
to  use  older  libraries  which  can  make  development  difficult.  It 
may  be  worth  trying  for  a  trial  period,  especially  if  you’re  unfa¬ 
miliar  with  Linux  systems.  It  also  has  a  number  of  useful  Python 
scripts  which  are  worth  reading  and  implementing  on  your  own  sys¬ 
tem.  We  have  since  installed  a  custom  cluster  configuration  running 
Ubuntu  Server  10.04  LTS  (Long  Term  Support) 

3.5  Study  Your  Space 

The  free-standing  structure  assumes  a  level  floor,  which  is  not  en¬ 
tirely  true  about  our  actual  space.  The  floor  is  cement  and  dips  a  bit 


at  the  opening  and  varies  across  the  surface.  As  a  result,  the  sup¬ 
ports  don’t  sit  very  level  and  it  was  harder  to  align  the  parts  of  the 
support  structure  on  uneven  ground,  which  led  to  minor  incongru¬ 
ousness  among  the  monitors.  While  it  is  only  barely  noticeable,  it 
could  likely  have  been  avoided  if  we  had  hired  a  flooring  contractor 
to  raise  and  level  the  floor  of  the  space  before  we  built  the  supports. 
Post-hoc  leveling  efforts  have  provided  no  improvement. 

3.6  Final  Thoughts 

The  bottom  line  is  that  you  should  make  the  best  use  of  your  space, 
but  you  will  need  to  exercise  restraint  in  your  design  to  provide 
room  for  maintenance  and  air  flow.  If  you  build  your  machines  into 
server-racks,  you  will  find  yourself  with  all  the  issues  related  to 
owning  a  server  room. 

Today,  there  are  a  number  of  companies  offering  turn-key  so¬ 
lutions  for  tiled-display  installations.  These  are  worth  comparing 
to  the  cost  of  designing  your  own  system  from  scratch.  Also  worth 
noting,  there  is  driver-level  support  in  AMD  Radeon  FirePro  graph¬ 
ics  cards  for  transparently  scaling  to  a  large  number  of  displays 
without  middleware  like  Equalizer,  this  feature  is  referred  to  as 
the  AMD  Eyefinity  Technology.  Nvidia  is  just  now  catching  up 
with  driver-level  support  for  scaling  to  3+  monitors,  but  currently 
AMD  shows  clear  advantage  in  this  space.  Avoiding  the  program¬ 
mer  overhead  of  designing  custom  tiled-display  applications  from 
the  ground  up  will  save  your  organization  the  most  in  the  long  run. 

4  Future  Work 

We  are  currently  engaged  with  demonstrating  the  cluster  with  pro¬ 
totypes  built  with  the  Equalizer  Framework.  We  plan  to  evaluation 
the  cluster  as  a  replacement  for  CAVE  applications  (although  it  is 
not  capable  of  stereoscopy),  as  well  as  collaborative  presentations 
and  command  center  applications.  We  also  plan  to  make  use  of  the 
space  for  the  study  of  perceptual  effects  related  to  the  field  of  view 
and  spatial  memory. 

Given  additional  budget  and  a  larger  space  for  the  tiled-display 
wall,  we  may  consider  removing  the  workstations  from  their  rack- 
mount  cases  and  placing  them  in  stand-alone  towers  that  could  be 
placed  near  the  wall  so  that  we  no  longer  need  to  go  though  the 
ceiling.  We  need  additional  space  for  easy  access  to  the  back  of  the 
monitor  matrix  for  maintenance  purposes.  If  the  machines  could  be 
placed  on  the  floor  behind  the  wall,  standard  DVI-D  cables  could 
reach  the  monitors.  While  it  is  elegant  to  have  the  machines  all 
stored  together  in  two  full  racks,  the  cables  must  be  longer  to  reach 
the  monitors,  even  if  the  racks  were  positioned  behind  the  wall. 
Although,  this  solution  would  not  be  perfect,  since  the  backs  of  the 
machines  will  face  the  back  of  the  tiled-display  wall,  and  the  nest 
of  display  cables  will  make  it  slightly  awkward  to  access. 
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